Papers with masked modeling
ScanEZ: Integrating Cognitive Models with Self-Supervised Learning for Spatiotemporal Scanpath Prediction (2025.acl-short)
Copied to clipboard
| Challenge: | ScanEZ framework provides a framework for predicting scanpaths during reading . masked modeling of eye movements and cognitive model simulations are used to kick-start training. |
| Approach: | They propose a framework for self-supervised learning that models scanpaths using synthetic data and a 3-D gaze objective inspired bymasked language modeling. |
| Outcome: | The proposed framework achieves state-of-the-art results on established datasets and is portable across different conditions. |
MMCLIP: Cross-Modal Attention Masked Modelling for Medical Language-Image Pre-Training (2026.acl-long)
Copied to clipboard
| Challenge: | Existing vision-and-language pretraining methods face challenges in reconstructing pathological features due to limited data. |
| Approach: | They propose a method that uses masked modeling to enhance visual and linguistic learning. |
| Outcome: | MMCLIP integrates unpaired data through disease-kind prompts to achieve state-of-the-art performance in zero-shot and fine-tuning across five benchmarks. |
Semantics-enhanced Cross-modal Masked Image Modeling for Vision-Language Pre-training (2024.lrec-main)
Copied to clipboard
Haowei Liu, Yaya Shi, Haiyang Xu, Chunfeng Yuan, Qinghao Ye, Chenliang Li, Ming Yan, Ji Zhang, Fei Huang, Bing Li, Weiming Hu
| Challenge: | Existing methods for vision-language pre-training lack high-level semantics and text is not sufficiently involved in masked modeling. |
| Approach: | They propose a semantics-enhanced cross-modal MIM framework for vision-language representation learning that harvests high-level semantics from global image features via self-supervised agreement learning and transfers them to local patch encodings by sharing the encode space. |
| Outcome: | The proposed model achieves state-of-the-art or competitive performance on multiple vision-language tasks. |